Review: Why is Perception Important?

• Goal: Communicate information visually
• Visualization technique maximize the information a viewer perceives
• Limits of human perception are a significant factor in understanding complex relationships
• Apply results of research on human perceptions for data visualization

Use Aesthetics to Improve Perception

• Use aesthetics to improve perception
• Take a very broad view of the term ‘aesthetic’
• A plot aesthetics is any property of a visualization which highlight aspects of the data relationships
• Aesthetics are used to project additional dimensions of complex data onto the 2-dimensional plot surface

Regression Lines

Regression lines draw viewers attention

• Typically use a nonlinear regression line
  • Polynomial
    
  • Splines - piece wise model
  • Lowess - local nonlinear regression
• Bootstrap confidence intervals show range of probable trends
  • More or bootstrap resampling later
• Ideally want strait line relationship
  • Nonlinear relationships often arise from non-Normal distributions
    
  • Linear relationship is more intuitive

Properties of Common Aesthetics

Visualizing Large Complex Data is Difficult

Problem: Modern data sets are growing in size and complexity

• Goal: Understand key relationships in large complex data sets

• Difficulty: Large data volume

  • Modern computational systems have massive capacity
    
  • Example: Use map-reduce algorithms on cloud clusters

• Difficulty: Large numbers of variables

  • Huge number of variables with many potential relationships
    
  • This is the hard part!

Note: we will address use of dimensionality reduction techniques in another lesson, time permitting

Limitation of Scientific Graphics

All scientific graphics are limited to a 2-dimensional projection

• But, complex data sets have a great many dimensions

• We need methods to project large complex data onto 2-dimensions

• Generally, multiple views are required to understand complex data sets

  • Don’t expect one view to show all important relationships
    
  • Develop understanding over many views
    
  • Try many views, don’t expect most to be very useful

Approaches to display of complex data relationships

Generally combine multiple methods to effectively display complex data

• Use plots that inherently scale

• Avoid over-plotting to ensure plot is understandable

• Choose plot types that do not exhibit overplotting

• Often a creative case specific plot type is best

• Use multi-axis plots

  • Scatter plot matricies
    
  • Facet plots

• Filter cases using cognositics

Scalable Chart Types

Some chart types are inherently scalable.

• Bar plots: Counts can be computed; e.g. use map-reduce
  
• Histograms: Data is binned in parallel
  
• Box plots: Finding the quartiles is a scalable counting process
  
• KDE and violin plots: Similarly to the box plot, using kernel density estimation

Over-plotting

Over-plotting occurs in plots when the markers lie one on another.

• Common, even in relatively small data sets
  
• Scatter plots can look like a blob and be completely uninterpretable
  
• Over-plotting is a significant problem in EDA and presentation graphics

Dealing with Over-plotting

What can we do about over-plotting?

• Marker transparency: so one can see markers underneath; useful in cases with minimal overlap of markers
  
• Marker size: smaller marker size reduces over-plotting within limits
  
• Adding jitter: adding a bit of random jitter to variables with limited number of values
• Random down-sampling: for very large data sets, you may only need a representative sample to understand key data relationships

Example of Overplotting

Use Transparency, Marker Size, Downsampling

Down sample to 20%, alpha = 0.1, size = 2

Other Methods to Display Large Data Sets

Alternatives to avoid over-plotting for truly large data sets

• Hex bin plots: the 2-dimensional equivalent of the histogram
  • Frequency of values is tabulated into 2-dimensional hexagonal bins
    
  • Displayed using a sequential color palette
• 2-d kernel density estimation plots: natural extension of the 1-dimensional KDE plot
  • Good for moderately large data
• Heat map: values of one variable against another
  • Categorical (count) or continuous variables
    
  • Carefully choose color pallet, sequential or divergent
• Mosaic plots: display multidimensional count (categorical) data
  • Uses tile size and color to project multiple dimensions
    
  • 2-d equivalent of a multivariate bar chart
• Dimensionality reduction: we will discuss this later in the course

Hexbin Plot

Example: Density of sale price by time

Countour Plot

Example: Contour plot of 2-D KDE of sale price vs. time

Heat Map

Example: Airline passenger counts by month and year displayed by squential heat map

Heat map of

Mosaic Plots

How can we display multidimensional count (categorical) data at scale?

• Mosaic plots displays the relative proportion of counts of a contingency table

• Plot area is divided and fully filled by a set of tiles

• The larger the count, the larger tile area

Mosaic Plots

Example: How do counts change with working vs. non-working day, weather and year?

• Mosaic plot displays tiles with size proportional to count

• Counts conditioned on variables, working day, weather, year

fig, ax = plt.subplots(figsize=(25, 20))
plt.rcParams.update({'font.size': 18})
_=mosaicplot.mosaic(bike_share_df.loc[:,categorical_cols], 
                    index=list(categorical_cols), 
                    title = 'Counts of weather conditions',
                    ax=ax)

plt.show()

Mosaic Plots

Facet plot of wind by month

Other Methods to Display Large Data Sets

Sometimes a creative alternative is best

• Often situation specific; many possibilities

• Finding a good one can require significant creativity!

• Example, choropleth for multi-dimensional geographic data

• Example, time series of box plots

Time Series of Box Plots

Example: Time ordered box plots of quarterly sales price

Displays for Complex Data

How can we understand the relationships in complex high-dimensional data with many variables?

• Must confront limitations for 2-dimensional projection inherent in computer graphics

• Need methods to scale to higher dimensions

• Apply combination of arrays of plots and filtering

• Aesthetics: used to increase the number of projected dimensions

Displays for Complex Data

How can we understand the relationships in complex high-dimensional data with many variables?

• Arrays of plots: subsets show relationships in a complex data set

• Pairwise scatter plots: matrix of all pairwise combinations of variables

  • pairwise scatter plots can be created for subsets of large and complex data sets.

• Faceting: uses values of categorical or numeric variables to plot subsets

  • Subsets are displayed on an array of plots
    
  • Typically use axes on same scale to ensure correct perception of relationships
    
  • Faceting goes by several other monikers, conditional plotting, method of small multiples, lattice plotting

• Cognostics: sort large number of variables to find important relationships

Arrays of Plots

Display multiple plot views in an array or grid

• Create an array of plots which project multiple related views of data relationships
  • Organize axes to give multidimensional view
• Example, scatterplot with kde plots on the margins
  • Supported by Seaborn jointplot

Scatter Plot Matrix

Scatter plot matrix used to investigate relationships between a number of variables

• Key idea: Display a scatter plots of each variable versus all other variables
  • Primarily EDA tool
    
  • Conveys lots of information - requires study!
• Each pairwise relationship is displayed twice
  • Two possible orientations
  • Or two different plot types
• Can place histograms and KDE plots on diagonal

Scatter Plot Matrix

Scatter plot matrix create two dimensional array of plots of variable pairs
- Upper triangular plots: Scatter plots with regression lines
- Lower triangular plots: Hexbin plots of density
- Diagonal plots: Histograms of variables

g = sns.PairGrid(diabetes.drop('Sex', axis=1), hue='sex_categorical', palette="Set2", height=1.5)
_=g.map_upper(sns.regplot, order=1, truncate=True, scatter_kws={'s':0.5})
_=g.map_lower(plt.hexbin, alpha=0.5, cmap='Blues', gridsize=15, linewi dths=0)
_=g.map_diag(plt.hist, histtype="step",linewidth=1)
plt.show()

Scatter Plot Matrix

Scatterplot matrix with different plot types

Facet Plots

Facet plots revolutionized statistical graphics starting about 30 years ago

• Facet plots extend the number of dimensions projected onto 2-d plot surface

• Key idea: Create array of plots of subsets of the data

  • Create subsets using a group-by operation on other variables
    
  • Lay out grid of plots on axes with same scale
    
  • Organize by row and column grouping variables
    
  • Display same plot type for each group
    
  • Can add specific aesthetics, etc.

Facet Plots

Like many good ideas facet plotting was invented several times

• Multiple contemporaneous inventors and names
  • Tufte, 1990, introduced method of small multiples
    
  • Cleveland, 1992, introduced trellis plotting
    
  • Also known as conditioned plots
• Nowdays, most packages use term facet plot

Facet Plot with wind speed by Month

Facet plot projected on a grid

• Grid defined by variable values for row and column
  

• Can tile or shingle numeric variables for grid

  • Tile categorical variables
    
  • Overlapping shingles of numeric variables
    

• Can create variables by combinations of categories from other variables

• Example: Histogram of wind speed conditioned on month

  • Plot grid has month in columns
    
  • Histogram of windspeed is created for each monthly subset

g = sns.FacetGrid(bike_share_df, col='month', col_order=calendar.month_abbr[1:] ,col_wrap = 4, height=5)
g = g.map(plt.hist, "windspeed", bins=20, color="b", alpha=0.5)

Facet Plot with wind speed by Month

Facet plot of wind by month

Facet Plot of Hourly Counts by Weather and Season

Example: Plot count of riders by hour, conditioned on weather and season

• Grid defined by season on the rows and weather in the columns
  
• Scatter plot of count of riders by hour of the data for each group

g = sns.FacetGrid(bike_share_df, col="Weather", col_order=weather.values(), row="Season", height=2, aspect=2)
g = g.map(sns.scatterplot, "hr", "cnt", s=3, alpha=0.2)
for ax in g.axes.flat:
    ax.set_title(ax.get_title(), fontsize=10)

Facet Plot of Hourly Counts by Weather and Season

Example: Plot count of riders by hour, conditioned on weather and season

Facet plot of count by season and weather

Facet Plot of Hourly Counts by Weather and Season

Example: Plot count of riders by hour, conditioned on weather and season

• Grid defined by season on the rows and weather in the columns
  
• Boxplots of count of riders by hour of the data for each group
  • Time-ordered box plots can improve perception over scatter plots
    
  • Eye tends to follow the lines

g = sns.FacetGrid(bike_share_df, col="Weather", col_order=weather.values(), row="Season", height=2, aspect=2)
g = g.map(sns.boxplot, "hr", 'cnt', color='lightgray', order=None)
for ax in g.axes.flat:
    ax.set_title(ax.get_title(), fontsize=10)
for i in range(4):
    g.axes[i,0].set_ylabel('Count') 

Facet Plot of Hourly Counts by Weather and Season

Example: Box plot counts of riders by hour, conditioned on weather and season

Facet plot of count by season and weather

Congnostics

How can we visualize very high dimensional data?

• Modern data sets have thousands to millions of variables

  • Cannot possibly look at all of these

• Idea: need to find the most important relationships

• Use a cognostic to sort relationship

  • Cognostic is a statistic to sort data
    
  • Sort the variables or relationships by the cognostic
    
  • Plot relationships with most interesting cognostic

• Idea originally proposed by Tukey, 1982, 1985

Cognistic: States With Fastest Rate of Housing Price Increase

## Add an intercept column to the data frame
housing.loc[:,'intercept'] = [1.0] * housing.shape[0]
## Demean the decimal time column
mean_time = housing.loc[:,'time_decimal'].mean()
housing.loc[:,'time_demean'] = housing.loc[:,'time_decimal'].subtract(mean_time)

## Find the slope coefficients for each state
def prepare_temp(df, group_value, group_variable = 'state'):
    temp = df.loc[df.loc[:,group_variable]==group_value,:].copy()
    mean_price = np.mean(temp.loc[:,'log_medSoldPriceSqft'])
    temp.loc[:,'log_medSoldPriceSqft'] = np.subtract(temp.loc[:,'log_medSoldPriceSqft'], mean_price)
    std_price = np.std(temp.loc[:,'log_medSoldPriceSqft'])
    temp.loc[:,'log_medSoldPriceSqft'] = np.divide(temp.loc[:,'log_medSoldPriceSqft'], std_price)
    return(temp, mean_price, std_price)
 

def compute_slopes(df, column, group_variable='state'):
    slopes = []
    entities = []
    intercepts = []
    for e in df.loc[:,column].unique():
        temp, mean_price, std_price = prepare_temp(df, e, group_variable=column)
        temp_OLS = sm.OLS(temp.loc[:,'log_medSoldPriceSqft'],temp.loc[:,['intercept','time_demean']]).fit()
        slopes.append(temp_OLS.params.time_demean)
        intercepts.append(temp_OLS.params.intercept)
        entities.append(e)
 
    slopes_df = pd.DataFrame({'slopes':slopes, 'intercept_coef':intercepts, 'entity_name':entities})    
    slopes_df.sort_values(by='slopes', ascending=False, inplace=True)
    slopes_df.reset_index(inplace=True, drop=True) 
    return slopes_df

#compute_slopes(housing, 'state')
state_slopes = compute_slopes(housing, 'state')

## PLot states with the fastest growing pricing
def find_changes(df, slopes, start, end, col='state'):
    increase  = slopes.loc[start:end,'entity_name']
    increase_df = df.loc[df.loc[:,col].isin(increase),:]
    increase_df = increase_df.merge(slopes, how='left', right_on='entity_name', left_on=col)
    return(increase_df, increase)
big_increase_states, increase_states = find_changes(housing, state_slopes, 0, 7)    

## Display scatterplot vs time
def plot_price_by_entity(df, order, entity='state', xlims=[2007.5, 2016.5]):
    g = sns.FacetGrid(df, col=entity, col_wrap=4, height=2, col_order=order)
    g = g.map(sns.regplot, 'time_decimal', 'log_medSoldPriceSqft', 
              line_kws={'color':'red'}, scatter_kws={'alpha': 0.1, 's':0.5})
    g.set(xlim=(xlims[0],xlims[1]))
    plt.show()

_=plot_price_by_entity(big_increase_states, increase_states)

Cognistic: States With Fastest Rate of Housing Price Increase

States with greatest increase in housing price

Summary

We have explored these key points

• Proper use of plot aesthetics enable projection of multiple dimensions of complex data onto the 2-dimensional plot surface.
  

• All plot aesthetics have limitations which must be understood to use them effectively
  

• The effectiveness of a plot aesthetic varies with the type and the application

• Visualization of modern data sets, growing in size and complexity

• Visualization limited by 2-dimensional projection

• Goal: Understand key relationships in large complex data sets

• Difficulty: Large data volume

  • Modern computational systems have massive capacity
    
  • Example: Use map-reduce algorithms on cloud clusters

• Difficulty: Large numbers of variables

  • Huge number of variables with many potential relationships
    
  • This is the hard part!

Summary

Generally combine multiple methods to effectively display complex data

• Use plots that inherently scale

• Avoid over-plotting to ensure plot is understandable

• Choose plot types that do not exhibit overplotting

• Often a creative case specific plot type is best

• Use multi-axis plots

  • Scatter plot matricies
    
  • Facet plots

• Filter cases using cognositics